This disclosure relates generally to algorithm development and more particularly to assessing the performance of an algorithm at different phases of development and assessing the impact that input variations have on the algorithm.
Generally, an algorithm undergoes several validations during the course of its development. Validation is the systematic evaluation of the algorithm performance. The evaluation includes performing reviews and tests to ensure that functional requirements are complete and testable. Generally, a review is a form of inspection that includes reviewing documentation to make sure that it supports operation, maintenance and future enhancements. In addition, the review includes determining whether established requirements, design concepts and specification have been met. Generally, a test includes operating the algorithm with real or simulated inputs to demonstrate that it satisfies the functional requirements. Typically, during a test, various test cases having selected input data and parameters are used as inputs to the algorithm. Outputs and reactions of the algorithms are then noted. Test failures are used to identify the specific differences between expected and actual results.
There are several drawbacks associated with the above-noted process for validating algorithms. Most notably, when the choice of algorithms to be implemented is not known a priori, for example because the algorithms are tasked with solving a new problem that has no known benchmark solution, the impact of newly developed parts of the algorithm on solving the stated problem cannot easily be judged. In addition, if the problem domain is poorly known, algorithmic solutions are carried without the benefit of knowing exactly the impact on the unknown parts of the problem domain. This predicament is compounded by several factors such as increasing complexity of the problem domain, multiple boundary conditions, and permutation of possible cases.
In order to overcome the above drawbacks, there is a need for a methodology that allows a software developer to assess the performance of an algorithm at different phases of development and assess the impact that input variations will have on the algorithm.